A general approach for automatic segmentation of pneumonia, pulmonary nodule, and tuberculosis in CT images

•GSAL reduces the workload of manual annotation lung lesions on CT images•The semantic dependencies in both spatial and channel dimensions are decoded•The self-supervised rotation loss mitigates discriminator forgetting GAN Proposing a general segmentation approach for lesions, including pulmonary nodules, pneumonia, tuberculosis, images will improve efficiency radiology. However, performance generative adversarial networks is hampered by limited availability annotated samples catastrophic discriminator, whereas universality traditional morphology-based methods insufficient segmenting diverse lesions. A cascaded dual-attention network with context-aware pyramid feature extraction module was designed to address these challenges. mitigate forgetting. The proposed model achieved Dice coefficients 70.92, 73.55, 68.52% multi-center nodule, tuberculosis test datasets, respectively. No significant decrease accuracy observed (p > 0.10) when small training sample size used. cyclic reduced < 0.01). promising multiple lesion types images. automatic images, COVID-19 pneumonia other similar findings, such as has always been challenging because various morphologies, intensities, locations lesions.1Song J. Huang S.-C. Kelly B. Liao G. Shi Wu N. Li W. Liu Z. Cui L. Lungre M. et al.Automatic nodule intra-nodular heterogeneity image generation.IEEE Biomed. Health Inform. 2022; 26: 2570-2581https://doi.org/10.1109/JBHI.2021.3135647Google Scholar Traditional methods, region-growing, level set, graph cut algorithms, have proven automatically segment certain lesions.2Song Yang C. Fan Wang K. F. S. Tian Lung using toboggan based growing approach.IEEE Trans. Med. Imaging. 2016; 35: 337-353https://doi.org/10.1109/TMI.2015.2474119Google Scholar,3Hoogi A. Beaulieu C.F. Cunha G.M. Heba E. Sirlin C.B. Napel Rubin D.L. Adaptive local window set MRI liver lesions.Med. Image Anal. 2017; 37: 46-55https://doi.org/10.1016/j.media.2017.01.002Google Scholar,4Nardelli P. Jimenez-Carretero D. Bermejo-Pelaez Washko G.R. Rahaghi F.N. Ledesma-Carbayo M.J. San Jose Estepar R. Pulmonary artery-vein classification deep learning.IEEE 2018; 2428-2440https://doi.org/10.1109/TMI.2018.2833385Google leakage regions, well over- under-segmentation limit different morphological variations. Emerging convolutional neural (CNN)-based particularly (GANs), demonstrated be effective architectures detection regions interest (ROI) medical images.5Ruan Y. Marshall H. Miao T. Cossetto Chan I. Daher O. Accorsi Goela MB-FSGAN: joint quantification kidney tumor multi-branch sharing network.Med. 2020; 64101721https://doi.org/10.1016/j.media.2020.101721Google Scholar,6Onishi Teramoto Tsujimoto Tsukamoto Saito Toyama Imaizumi Fujita Multiplanar analysis networks.Int. Comput. Assist. Radiol. Surg. 15: 173-178https://doi.org/10.1007/s11548-019-02092-zGoogle Scholar,7Qin Zheng X. Zhu Y.M. synthesis networks.Med. Phys. 2019; 46: 1218-1229Google Scholar,8Valvano Leo Tsaftaris S.A. Learning from scribbles multi-scale attention gates.IEEE 2021; 40: 1990-2001Google Continuous collaborative generator enabled detect characteristics produce synthetic directly extracting ROIs source Previous studies that an segmentation. Zhang al. conditional region pneumonic simultaneously pyramid-based architecture.9Zhang Zhong Deng Tang CoSinGAN: learning infection single radiological image.Diagnostics. 10901https://doi.org/10.3390/diagnostics10110901Google ROI masks at scales were used process via mimic better resolution, average dice coefficient (DC) 54.1% standard deviation 21.6% obtained. In another study, weakly supervised architecture proposed, consisting producing mask, replacing predicted generated resembles uninfected area, distinguish healthy real regions.10Wang Fu Q. Zhou Feng Ma weakly-supervised framework localization chest CT.IEEE 39: 2615-2625https://doi.org/10.1109/TMI.2020.2995965Google By successfully deceiving DC 70.3% Second, three-dimensional (3D) GAN-based data augmentation bounding box-based 3D detection.11Han Kitamura Kudo Ichinose Rundo Furukawa Umemoto Nakayama Synthesizing nodules wherever massively: multi-conditional object detection.arXiv. (Preprint at)https://doi.org/10.48550/arXiv.1906.04962Google two discriminators applied classify pairs nodules. addition, image-style transfer synthesize robust segmentation.12Shi Lu Novel Data Augmentation Method Using Style-Based Robust Nodule Segmentation. IEEE), 2020: 2486-2491Google Scholar,13Gao Clark Furst Raicu Augmenting LIDC Dataset Generative Adversarial Networks Improve Detection. International Society Optics Photonics), 2019: 109501KGoogle aforementioned studies, DC>80% obtained augmented datasets Database Consortium collection (LIDC-IDRI). integrating multi-omics data, end-to-end fusion gene sequencing ensure authenticity quality synthesized images.14Xu Shin H.C. Roth Milletari Xu Correlation Synthesis: End-To-End Generation Radiogenomic Map Based Network.2019Google Furthermore, although admission symptoms those it reported responsible more than million deaths per year World Organization Global Tuberculosis Report. intensity changes pneumonia.15World OrganizationGlobal Report 2013. Organization, 2013Google One previous nine CNNs X-ray images.16Rahman Khandakar Kadir M.A. Islam K.R. K.F. Mazhar Hamid M.T. Kashem Mahbub Z.B. al.Reliable learning, visualization.IEEE Access. 8: 191586-191601Google affinity metric clustering select optimal thresholding spatially diffuse multi-focal radiotracer uptake PET sensitivity specificity higher 90%.17Foster Bagci U. Ziyue Dey Luna Bishai Jain Mollura D.J. Segmentation computer-aided functional animal models.IEEE Eng. 2014; 61: 711-724https://doi.org/10.1109/TBME.2013.2288258Google U-net architecture, (CXR) modality-specific U-Nets ROIs, 75.5% VGG16-CXR-U-Net architecture.18Rajaraman Folio L.R. Dimperio Alderson P.O. Antani S.K. Improved tuberculosis-consistent findings X-rays models weak localizations.Diagnostics. 11616https://doi.org/10.3390/diagnostics11040616Google Recently, disentangle area detection,19Han Lyu Peng GAN-Based Disentanglement Chest X-Ray Rib Suppression. Medical Analysis, 2022102369Google which intersection-over-union 60.3% CXR Despite tasks, development GAN-derived three drawbacks. First, manually required achieve precise Because there no universal solution large dataset labeled specific each study guarantee diversity. limitations time labor resources, inter-observer bias. For example, takes approximately 30 s experienced radiologist delineate boundary solitary diameter 10 mm one slice; this delineation procedure even longer adhesion-type or cases most mature trained hundreds thousands ImageNet.20Deng Dong Socher L.-J. Fei-Fei Imagenet: Large-Scale Hierarchical Database. Ieee), 2009: 248-255Google unlike natural often difficult obtain patient sources privacy regulations institutions. Although delineations radiologists 1,018 patients published LIDC-IDRI dataset21Armato 3rd, S.G. McLennan Bidaut McNitt-Gray M.F. Meyer C.R. Reeves A.P. Zhao Aberle D.R. Henschke C.I. Hoffman E.A. al.The database (LIDC) resource initiative (IDRI): completed reference scans.Med. 2011; 38: 915-931https://doi.org/10.1118/1.3528204Google radiologists’ annotations available,22Ma An Ge Yu Chen He Towards efficient covid-19 ct annotation: benchmark segmentation.arXiv. at)https://doi.org/10.48550/arXiv.2004.12537Google Scholar,23Morozov Andreychenko Pavlov Vladzymyrskyy Ledikhova Gombolevskiy V. Blokhin I.A. Gelezhe Gonchar Chernina V.Y. Mosmeddata: scans related dataset.arXiv. at)https://doi.org/10.48550/arXiv.2005.06465Google they current CNN millions parameters, latest GANs. information emphasized map layers through step-by-step convolution varies significantly. Specifically, features global semantics, relevant precisely locating salient distributed deeper layers. Conversely, shallow layer rich texture information, appropriate representing details target regions.24Zhao Pyramid saliency at): 3085-3094https://doi.org/10.48550/arXiv.1903.00179Google Studies benefits adaptive integration maps corresponding dependencies.25Fu Bao Fang Dual scene 3146-3154https://doi.org/10.48550/arXiv.1809.02983Google Scholar,26Gao Yuan SCAR: spatial-/channel-wise regression crowd counting.Neurocomputing. 363: 1-8Google encoding decoding consecutively implemented without distinction algorithms.27Zhao Han Jia Gou Detection U-Net Contextual Convolutional Neural Network. 2018: 356-361Google Scholar,28Wang Gu Zang Gevaert Central focused networks: developing data-driven segmentation.Med. 172-183Google Therefore, reorganizes recognizes texture, shape, location invariance would helpful segmentation.29Lin T.-Y. Dollár Girshick Hariharan Belongie Feature 2117-2125https://doi.org/10.48550/arXiv.1612.03144Google Finally, alternating stochastic gradient descent scheme causes depend output.30Schonfeld Schiele Khoreva u-net networks.arXiv. 8207-8216https://doi.org/10.48550/arXiv.2002.12655Google Scholar,31Chen Zhai Ritter Lucic Houlsby Self-supervised gans auxiliary loss.arXiv. 12154-12163https://doi.org/10.48550/arXiv.1811.11212Google classifier distribution class (i.e., fake samples) shifts during training. case, disregards learned produced generator, nonstationary environments.32Jaiswal Moyer Ver Steeg AbdAlmageed Natarajan Invariant representations forgetting.arXiv. 4272-4279https://doi.org/10.48550/arXiv.1911.04060Google Training may become unstable if cannot recall boundaries segmentations. This affects eventually mimicked image. emerging shown potential decode textural reduce instability field computer vision, mechanism focuses discriminating object-tracking tasks.33Wang Gupta Non-local 7794-7803https://doi.org/10.48550/arXiv.1711.07971Google Scholar,34Zhang Goodfellow Metaxas Odena Self-attention networks. PMLR), 7354-7363Google Scholar,35Wang Teng Xing Gao Hu Maybank Attentions: Residual Attentional Siamese Network High Performance Online Visual Tracking.2018: 4854-4863Google analysis, classification,36Woo Park Lee J.-Y. So Kweon Cbam: block module.arXiv. 3-19https://doi.org/10.48550/arXiv.1807.06521Google detection,37Khan F.S. van de Weijer Vanrell Modulating shape color recognition.Int. Vis. 2012; 98: 49-64Google segmentation.25Fu Among these, modules widely emphasizes “what” informative part should focus “where” located. necessary contributions do not equally represent object, some channels visual pattern others, vice-versa. Each high-level can regarded particular pattern, patterns associated other.38Rahman M.M. Fiaz Jung Efficient tracking stacked channel-spatial learning. IEEE Access, 2020Google aim optimize channel-wise weights around segmented.39Zhang super-resolution very residual 286-301https://doi.org/10.48550/arXiv.1807.02758Google exploiting interdependencies between maps, interdependent representation semantics regions. encodes wider range contextual into features, thereby enhancing their capabilities. It highlights so determined. Thus, complements attention.40Wang Jiang Qian classification.arXiv. 3156-3164https://doi.org/10.48550/arXiv.1704.06904Google Pooling type selects probable attention-weighted features.41Xu Ba Kiros Cho Courville Salakhudinov Zemel Bengio Show, attend tell: caption generation attention.arXiv. 2015; 2048-2057https://doi.org/10.48550/arXiv.1502.03044Google Scholar,42Zhu Groth Bernstein Visual7w: grounded question answering images.arXiv. 4995-5004Google addition basic forms attention, variations network,43Xu Saenko Ask, Attend Answer: Exploring Question-Guided Spatial Attention Question Answering. Springer, 2016: 451-466Google later attention-based modulated earlier one, help locate desired applies perform same task proposed.38Rahman Scholar,44Chen Xiao Nie Shao Chua T.-S. Sca-cnn: captioning.arXiv. 5659-5667https://doi.org/10.48550/arXiv.1611.05594Google constantly updated training, input changed accordingly. dynamic situation, arises continuously extract new input. Discriminator contributes instability.32Jaiswal To problem, GAN, allows access feasible approaches avoiding forgetting.45Odena Olah Shlens Conditional GANs.arXiv. at)https://doi.org/10.48550/arXiv.1610.09585Google Scholar,46Miyato Koyama cGANs projection discriminator.arXiv. at)https://doi.org/10.48550/arXiv.1802.05637Google integrates stimulate collaboration while competing synthesis.31Chen pretext task, predicting rotated angle relative patch, then resulting network. loss, retains generalizable environment, prevents classes iterations.47Doersch Efros A.A. Unsupervised context prediction.arXiv. 1422-1430https://doi.org/10.48550/arXiv.1505.05192Google Scholar,48Gidaris Singh Komodakis rotations.arXiv. at)https://doi.org/10.48550/arXiv.1803.07728Google enhance rare. summary, above-mentioned challenges loss. we first herein dual-feature Compared main advantages overcome segmentation.1)Reducing annotating training.2)Decoding efficiency.3)Mitigating preserving remainder paper structured follows. “results” section outlines experiments performed develop comparison made state-of-the-art methods. Followed “discussion”, “conclusions”, “limitations study” paper. “STAR Methods” describes detail. Table 1 presents results DCs 77.63, 75.01, 74.93% all (M_100) validation, Similarly, M_40 (training 40%) 72.01, 71.33, 70.92%, M_10 10%) 63.20, 59.39, 58.55%, difference found only 10%, 1,166 finding indicates several thousand GSAL tens significantly effort future public health emergencies, pneumonia.Table 1Pneumonia datasetsDCTrainingValidationTestp valueM_10077.63%75.01%74.93%Ref.M_7076.96%73.77%72.19%0.509M_4072.01%71.33%70.92%0.220M_1063.20%59.39%58.55%<0.01M_100, M_70, M_40, 100%, 70%, 40% 10% Open table tab M_100, 2 lists non-COVID-19 showed segmentation; however, 0.2) (see 2). demonstrate enables community-acquired pneumonia. applicability generic algorithm Figure shows example manifestation bilateral, mixed ground-glass opacity (GGO) consolidation, Figures S1–S3 show subtypes: https://github.com/JD910/general_net_for_lesion_seg#supp_materials.Table 2DC datasetDCCOVID-19non-COVID-19p valueM_10077.36%72.68%0.410M_7073.55%70.03%0.353M_4071.30%67.11%0.341M_1060.80%57.17%0.208 3 82.39, 77.66, 76.30% M_100 76.10, 73.90, 73.55% M_10. = 0.209). segmentation, 75%, studies.Table 3Lung valueM_10082.39%77.66%76.30%Ref.M_7080.55%76.62%76.23%0.685M_4077.91%75.63%74.10%0.566M_1076.10%73.90%73.55%0.209 4 subtype dataset. solid, juxta-vascular, juxta-pleural, GGO subtypes identified clinical experts. revealed solid